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Administrivia 

• PS 4: Out and not changed. Due Sunday Oct 
12 th , 11:55pm 

• It is application of the last few lectures. Mostly straight forward 
Matlab but if you’re linear algebra is rusty it can take a while to 
figure out. Question 2 takes some doing to understand. 

• You have been warned... 

• It is cool 

• You have been warned... 

• Today: Start on features. 

• Forsyth and Ponce: 5. 3-5. 4 

• Szeliski also covers this well - Section 4 - 4.1.1 

• These next 3 lectures will provide detail for Project 4. 
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The basic image point matching problem 

• Suppose I have two images related by some 
transformation. Or have two images of the same object in 
different positions. 

• How to find the transformation of image 1 that would align 
it with image 2? 
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We want Locala) Features® 

• Goal: Find points in an image that can be: 

• Found in other images 

• Found precisely - well localized 

• Found reliably - well matched 

• Why? 

• Want to compute a fundamental matrix to recover geometry 

• Robotics/Vision: See how a bunch of points move from one frame 
to another. Allows computation of how camera moved -> depth -> 
moving objects 

• Build a panorama... 
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Suppose you want to build a panorama 



M. Brown and D. G. Lowe. Recognising Panoramas. ICCV 2003 
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How do we build panorama? 

• We need to match (align) images 
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Matching with Features 

• Detect features (feature points) in both images 
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Matching with Features 

• Detect features (feature points) in both images 

• Match features - find corresponding pairs 
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Matching with Features 

• Detect features (feature points) in both images 

• Match features - find corresponding pairs 

• Use these pairs to align images 




CS 4495 Computer Vision - A. Bobick 


Features 1 : Harris 
and other corners 


Matching with Features 

• Problem 1: 

• Detect the same point independently in both images 



no chance to match! 


We need a repeatable detector 
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Matching with Features 

• Problem 2: 

• For each point correctly recognize the corresponding one 



We need a reliable and distinctive descriptor 
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More motivation . . . 


• Feature points are used also for: 

• Image alignment (e.g. homography or fundamental 
matrix) 

• 3D reconstruction 

• Motion tracking 

• Object recognition 

• Indexing and database retrieval 

• Robot navigation 

• ... other 
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Characteristics of good features 


• Repeatability/Precision 

• The same feature can be found in several images despite geometric 
and photometric transformations 

• Saliency/Matchability 

• Each feature has a distinctive description 

• Compactness and efficiency 

• Many fewer features than image pixels 

• Locality 

• A feature occupies a relatively small area of the image; robust to 
clutter and occlusion 
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Corner Detection: Basic Idea 


• We should easily recognize the point by looking through a 
small window 


• Shifting a window in any direction should give a large 
change in intensity 



“flat” region: 
no change in 
all directions 



“edge”: 
no change 
along the edge 
direction 



“corner”: 

significant change 
in all directions with 
small shift 


Source: A. Efros 
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Finding Corners 



• Key property: in the region around a corner, image 
gradient has two or more dominant directions 

• Corners are repeatable and distinctive 


C. Harris and M. Stephens. "A Combined Corner and Edge Detector/' 
Proceedings of the 4thAlvey Vision Conference : pages 147 — 151, 1988 
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Finding Harris Comers 



• Key property: in the region around a corner, image 
gradient has two or more dominant directions 

• Corners are repeatable and distinctive 


c. Harris and M. Stephens. "A Combined Corner and Edge Detector/' 
Proceedings of the 4thAlvey Vision Conference : pages 147 — 151, 1988 
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Change in appearance for the shift [u,v]\ 


E(u,v) = Y J w(x,y)[i( 

x V \ 

.V + u, y + v) - f ' 

(x, y)] 2 

' / ' 

1 




\ function J 


intensity 


ntensity 


Window function w(x,y) - 



1 in window, 0 outside Gaussian 


Source: R. Szeliski 
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Corner Detection: Mathematics 

Change in appearance for the shift [u,v]\ 

E(u,v) = ^w(x,y)[l(x + u,y + v)-J(x,y)] 



E(u, v) 
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Corner Detection: Mathematics 

Change in appearance for the shift [u,v]\ 

E(u,v) = 2^w(x, y)[l(x + u,y + v)-I(x, y)] 

*>y 

We want to find out how this function behaves for 
small shifts (u,v near 0,0) 
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Corner Detection: Mathematics 

Change in appearance for the shift [u,v]\ 


E(u,v) = ^w(x,y)[l(x + u,y + v)-J(x,y)] 


Second-order Taylor expansion of E(u,v ) about (0,0) 
(local quadratic approximation for small u,v)\ 


(lD):F(dx)«F(0) + dx- 


dF( 0) 1 _ 2 d 2 F( 0) 


dx 


+—Sx 

2 


dx 


E(u,v ) « E(0,0) + [u v] 


E„(0,0) 

+ -[u V] 

£,. ( 0. 0 ) _ 

2 


E uu (0,0) E uv (0, 0) 


F uv (0,0) E w (0, 0) 


U 


V 
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Co rner Detection: Mathematics 

E(u,v) = ^w(x,y)[l(x + u,y + v)-J(x,y)] 

x,y 


Second-order Taylor expansion of E(u,v) about (0,0) 


E(u,v ) « E(0,0) + [u v] 


X (0,0)' 

+ -[u v] 

_T V (0,0)_ 

2 


E uu (0,0) E uv (0,0) 
E(0,0) E (0,0) 


U 


V 
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Corner Detection: Mathematics 



Second-order Taylor expansion of E(u,v ) about (0,0) 


E(u,v ) « E(0,0) + [u v] 


X (0,0)' 

+ -[u V] 

A(o,o)_ 

2 


E (0,0) E (0,0) 


E uv (0,0) E vv (0,0) 


U 


V 


E u ( u > v ) = Z 2 w ( x > yt 1 ( x + u > y + v ) - 1 (*» y)K (x + u, y + v) 

*>y 

e uu ( u > v ) = Z 2w (*> y)T(* + y + v)T(* + y + v) 

*>y 

+ ^ 2w(x, y)[l(x + u, y + v) - I(x, y)]j M (x + u,y + v) 

*,y 

E uv (u, v) = Z 2 w(x, y)J y (x + u, y + v) J x (x + u, y + v) 

*>y 

+ Z 2 w ( x > y )[ 7 (x + u, y + v) - 1 (x, y)]/ x> , (x + u, y + v) 

*>y 
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Corner Detection: Mathematics 



Second-order Taylor expansion of E(u,v ) about (0,0) 


E(u,v ) « E(0,0) + [u v] 


X (0,0)“ 

+ -[u v] 

_E V ( 0,0)_ 

2 


E (0,0) E (0,0) 


E uv (0,0) E vv (0,0) 


U 


V 


E u ( u > v ) = Z 2 w ( x > yt 1 ( x + u > y + v ) - 1 (*» y)K (x + u, y + v) 

*>y 

E uu ( u > v) = Z 2 w (x, yVx (x + u, y + v)/ x (x + u, y + v) 

*>y 

+ Y J 2w(x,y)[l(x + u,y + v)-I(x,y)][ xx (x + u,y + v) 

*,y 

E uv (u,v) = Y J 2w(x, y)I y ( X + U,y + v)I x (x + u, y + v) 

+ Y 2 w ( x > y)[^ (x + u, y + v) - 7 (x, y)]l xy (x + u, y + v) 

*>y 
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Corner Detection: Mathematics 



Second-order Taylor expansion of E(u,v ) about (0,0) 


E(u,v ) « E(0,0) + [u v] 


X (0,0)' 

+ -[u V] 

A(o,o)_ 

2 


E (0,0) E (0,0) 


E uv (0,0) E vv (0,0) 


U 


V 


E u ( u > v ) = Z 2 w ( x > yt 1 ( x + u > y + v ) - 1 (*» y)K (x + u, y + v) 

*>y 

E uu ( u > v) = Z 2 w (x, yVx (x + u, y + v)/ x (x + u, y + v) 

*>y 

+ Y J 2w(x,y)[l(x + u,y + v)-I(x,y)][ xx (x + u,y + v) 

*,y 

E uv (u, v) = 2 w(x, y)J y (x + u, y + v)I x (x + u, y + v) 

*>y 

+ £ 2 w(x, y)[/(x + u, y + v) - 1 (x, y)]/^ (x + u, y + v) 

*>y 
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Corner Detection: Mathematics 



Evaluate at (u,v) 


( 0 , 0 ): 


E(u, v) 






E(0,0) 

+ [u v] 

(0,0) 

£ v (0,0) 

+ ^[U V] 
2 


E (0,0) E (0,0) 


E uv (0,0) E vv (0,0) 


vv 


E u (m > v ) = Z 2 w ( x ’ yt 1 ( x u ’ y + v ) - 1 (*, y)K (* + y + v ) 




0 


E uu (u, v) = y 2 w(x, y)J x (x + u, y + v) J x (x + u, y + v) 


*>y 


+ 


^2w(x,y)p(x + »,y + v)-J(x,y) f xx (x + »,y + v) 


*,y 


0 


E uv (u, v) = Y 2 w(x, y)I y (x + u,y + v)7 x (x + u,y + v) 


*>y 


*,y 


+ y 2w(x, y)[j'(x + u, y + v) - / (x, y)]f (x + u, y + v) 


0 


u 


V 
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Corner Detection: Mathematics 



Second-order Taylor expansion of E(u,v ) about (0,0) 


£(u,v) « £(0,0) + [u v] 


E u ( 0 , 0 ) 

Ejm 



E( 0,0) = 0 E uu (0, 0) = X 2 w(x, y)I x (x, y)I x (x, y) 

E (0, 0) = 0 E w (0, 0) = 1 2 w(x, y)I y (x, y)I y (x, y) 

*,y 

E (0, 0) = 0 E v (0, 0) = I 2 w(x, y ) J (x, y) J (x, y) 
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Co rner Detection: Mathematics 

E(u,v) = 2^w(x, y)[l(x + u,y + v)-I(x, y)] 

*>y 

Second-order Taylor expansion of E(u,v ) about (0,0): 



2 w ( x > y)T 2 (*» y) 

^w(x,y)I x (x,y)I y (x,y) 

I I 

£(i/,v)«[i/ v] 

^ w(x, y)/ x (x, y)7 y (x, y) 

_ x,y 

£ w(x, y)7 2 (x, y) 

u 

V 


E(0,0) = 0 E u (0,0) = I2w(x,y)J (x,y)J (x,y) 

x,y 

E (0, 0) = 0 E n (0, 0) = I 2 w(x, y)I ( x , y)I (x, y ) 

x,y 

E (0, 0) = 0 E v (0, 0) = 1 2w(x, y)I (x, y)I (x, y) 
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Corner Detection: Mathematics 

The quadratic approximation simplifies to 


E(u,v ) « [u v] M 



where M is a second moment matrix computed from image 
derivatives: 


Without 
weight 
M = 


m = Yj w ( x > y) 

i 2 

1 1 " 

X 

1 1 

* y 

i 2 

x >y 

L * y 

y J 


/ Ixlx Ixly 

T, Ixly y lyly 


E 


I 


x 


i 


y 


Each product is 
a rank 1 2x2 


[lx Iy ] = E V/(V/) 


T 
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Interpreting the second moment matrix 


The surface E(u,v ) is locally approximated by a 
quadratic form. 


u 


E(u,v ) « [u v] M 


M=Yj w ( x , y) 

*>y 
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Interpreting the second moment matrix 


Consider a constant “slice” of E(u, v): [u v] M 

I 2 u 2 + 2 IJ v uv + /V = k 

x x y y 

This is the equation of an ellipse. 


u 

v 


= const 
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Interpreting the second moment matrix 


First, consider the axis-aligned case 
where gradients are either horizontal or 
vertical 



If either A is close to 0, then this is not a corner, so 
look for locations where both are large. 
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Interpreting the second moment matrix 


First, consider the axis-aligned case 
where gradients are either horizontal or 
vertical 



If either A is close to 0, then this is not a corner, so 
look for locations where both are large. 
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Interpreting the second moment matrix 


Consider a horizontal “slice” of E(u, v): [u v] M 


u 


v 


This is the equation of an ellipse. 


Diagonalization of M: 


M = R 


-1 


\ 0 

0 /L 


R 


The axis lengths of the ellipse are determined by the 
eigenvalues and the orientation is determined by R 


direction of the 



direction of the 
slowest change 


const 
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Interpreting the eigenvalues 


Classification of image points using eigenvalues 
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Harris corner response function 

R = det(M)-atrace(M) 2 = \A 2 -a{\ + A 2 ) 2 

a: constant (0.04 to 0.06) 

• R depends only on 
eigenvalues of M, but 
don’t compute them (no 
sqrt, so really fast! 

• R is large for a corner 

• R is negative with large 
magnitude for an edge 

• |f?| is small for a flat 
region 
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Low texture region 



£v/(V/) r 

— gradients have small magnitude 

- small small k 2 




12 
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£v/(v/) r 

— large gradients, all the same 

- large X v small X 2 
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High textured region 


50 


100 


150 


200 



100 


150 


200 


250 


300 


350 


£v/(w) 


— gradients are different, large magnitudes 

- large X v large X 2 


s 10 


15 
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Harris detector: Algorithm 

1. Compute Gaussian derivatives at each pixel 

2. Compute second moment matrix M in a Gaussian 
window around each pixel 

3. Compute corner response function R 

4. Threshold R 

5. Find local maxima of response function (nonmaximum 
suppression) 


C. Harris and M. Stephens. "A Combined Corner and Edge Detector/' 
Proceedings of the 4thAlvey Vision Conference : pages 147 — 151, 1988. 
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Harris Detector: Workflow 
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Harris Detector: Workflow 

Compute comer response R 
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Features 1 : Harris 
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Find points with large corner response: i?>threshold 
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Harris Detector: Workflow 


Take only the points of local maxima of R 
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Harris Detector: Workflow 
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Other corners: 

• Shi-Tomasi ’94: 

• “Cornerness” = min (k v k 2 ) Find local maximums 

• cvGoodFeaturesToTrack(...) 

• Reportedly better for region undergoing affine deformations 



det M 

• Brown, M., Szeliski, R., and Winder, S. (2005): ^ + ~ 

• there are others... 
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Harris Detector: Some Properties 
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Harris Detector: Some Properties 


• Rotation invariance? 
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Harris Detector: Some Properties 


• Rotation invariance 


N 






Ellipse rotates but its shape (i.e. eigenvalues) remains 
the same 
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Harris Corner Detector 

Repeatability rate: 

# correspondences 
# possible correspondences 




C. Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000 



CS 4495 Computer Vision - A. Bobick 


Features 1 : Harris 
and other corners 


Harris Detector: Some Properties 

• Invariance to image intensity change? 



CS 4495 Computer Vision - A. Bobick 


Features 1 : Harris 
and other corners 


Harris Detector: Some Properties 

• Partial invariance to additive and multiplicative intensity 
changes (threshold issue for multiplicative) 

'C Only derivatives are used => invariance 
to intensity shift I — > I + b 


■C Intensity scale: I — » a I 
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Harris Detector: Some Properties 

• Invariant to image scale? 
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Harris Detector: Some Properties 


• Not invariant to image sca/e! 



All points will be 
classified as edges 


Corner ! 






repeatability rate 
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Harris Detector: Some Properties 

• Quality of Harris detector for different scale changes 
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*/F* we want scale invariance. 
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Scale Invariant Detection 

• Consider regions (e.g. circles) of different sizes around a 
point 

• Regions of corresponding sizes will look the same in both 
images 
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Scale Invariant Detection 

• The problem: how do we choose corresponding circles 
independently in each image? 
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• Solution: 

• Design a function on the region (circle), which is “scale invariant” 
(the same for corresponding regions, even if they are at different 
scales) 

Example: average intensity. For corresponding regions 
(even of different sizes) it will be the same. 


For some given point in one image, we can consider it 
as a function of region size (circle radius) 



scale = 1/2 
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• Common approach: 


Take a local maximum of this function 


Observation: region size, for which the 
maximum is achieved, should be invariant to 
image scale. 

Important: this scale invariant region size is 
found in each image independently! 



scale = 1/2 
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Scale Invariant Detection 

• A “good” function for scale detection: 
has one stable sharp peak 



region size region size region size 


• For usual images: a good function would be a one 
which responds to contrast (sharp local intensity 
change) 
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Scale sensitive response 





2.0 

10 . 1 

19. 2.0 3.89 

i 




scale 
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Scale Invariant Detection 


Function is just application of a kernel: f = Kernel * Image 

(Laplacian of Gaussian - LoG) 

L = <J y, O') + G yy (x, y, c)^ Laplacian of Gaussian 






CS 4495 Computer Vision - A. Bobick 


Features 1 : Harris 
and other corners 


Scale Invariant Detection 


• Functions for determining scale 

f = Kernel * Image 


Kernels: 

L = (J 2 (G xx (x, y, cr) + G yy (x, y, cr)) 

(Laplacian) 

DoG = G(x, y, kcr) - G(x, y, a) 

(Difference of Gaussians) 

where Gaussian 



2 2 
X +y 


G C,y^) = -ji^e ^ 


Note: both kernels are invariant to 
scale and rotation 
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Key point localization 

• General idea: find 
robust extremum 
(maximum or 
minimum) both in 
space and in scale. 


Features 1 : Harris 
and other corners 
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Key point localization 

•SIFT: Scale Invariant 
Feature Transform 

•Specific suggestion: 
use DoG pyramid to 
find maximum values 
(remember edge 
detection?) - then 
eliminate “edges” and 
pick only corners. 


Features 1 : Harris 
and other corners 
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Key point localization 


(Each point is compared 
to its 8 neighbors in the 
current image and 9 
neighbors each in the 
scales above and 
below. ) 


Features 1 : Harris 
and other corners 
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Scale space processed one octave at a time 


Scale 

(next 

octave) 




Scale 

(first 

octave) 





Gaussian 


Difference of 

Gaussian (DOG) 
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Extrema at different scales 
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Remove low contrast, edge bound 


+ V.-v* £f«* + 4 + /: + 

*■♦♦£**£ +£+** > V + 


imm + + j 
v+ .4 

* i. +X4fii+ . +.+ 



Extrema points 



Contrast > C 



Not on edge 





CS 4495 Computer Vision - A. Bobick 


Features 1 : Harris 
and other corners 


Scale Invariant Detectors 


SIFT (Lowe ) 1 

Find local maximum of: 

- Difference of Gaussians in 
space and scale 



• Harris-Laplacian 2 

Find local maximum of: 

• Harris corner detector in 
space (image 
coordinates) 

• Laplacian in scale 


Harris — » 





1 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004 

2 K.Mikolajczyk, C. Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001 
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Scale Invariant Detectors 


• Experimental evaluation of detectors 
w.r.t. scale change 


Repeatability rate: 

# correspondences 

# possible correspondences 




K.Mikolajczyk, C. Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001 
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Scale Invariant Detection: Summary 

• Given: two images of the same scene with a large scale 
difference between them 

• Goal: find the same interest points independently in each 
image 

• Solution: search for maxima of suitable functions in scale 
and in space (over the image) 

Methods: 

1. SIFT [Lowe]: maximize Difference of Gaussians over scale and 
space 

2. Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian 
over scale, Harris' measure of corner response over the 
image 
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Point Descriptors 

• We know how to detect points 

• Next question: How to match them? 



Point descriptor should be: 

1 . Invariant 

2. Distinctive 
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Next time 

• SIFT, SURF, 


iobick 
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SFOP, oh my... 



